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Abstract: Peer Instruction (Pl) is a student-centric pedagogy in which students move from the role of passive listeners to 
active participants in the classroom. Over the past five years, there have been a number of research articles regarding the 
value of Pl in computer science. The present work adds to this body of knowledge by examining outcomes from seven 
introductory programming instructors: three novices to PI and four with a range of Pl experience. Through common 
measurements of student perceptions, we provide evidence that introductory computing instructors can successfully 
implement PI in their classrooms. We find encouraging minimum (74%) and average (92%) levels of success as measured 
through student valuation of PI for their learning. This work also documents and hypothesizes reasons for comparatively 
poor survey results in one course, highlighting the importance of the choice of grading policy (participation vs. 
correctness) for new PI adopters. 


I. INTRODUCTION 

Peer Instruction (PI) has gained considerable traction among computer science educators and there have been a number 
of studies demonstrating its efficacy in a variety of dimensions. Students value PI [5, 8, 10] and learn more in PI classes 
compared to traditional lecture classes [11,13]. Pl is also associated with low failure rates [6] and increased retention of 
majors[9]. The vast majority of Pl studies in CS take the form of evaluating a single instructor [5,8,10] or implementation 
at a single institution [5, 6, 9]. As such, one concern is that the reported PI results over represent those occasions where 
PI Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without 
fee provided that copies are not made or distributed for profit or commercial advantage and that copies berth is notice 
and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be 
honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to 
lists, requires prior specific permission and/or a fee. Has proven successful. Given the current trend of increased PI 
adoption, it is important to establish the kinds of outcomes that can be expected across larger datasets and institution 
types. In addition, it is important to begin studying the ways in which new adopters adopt PI. To what extent is Pl adopted 
wholesale? How are the steps of PI altered to suit the instructor or student demands? 
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This paper reports on a set of Pl adoptions with a broad range of class parameters and types of institutions. Only one 
other paper has offered such a multi-institution view, but it examined only small classes at private liberal-arts colleges 
[8].In the present study, seven instructors and their students, from multiple institutions of different types, were surveyed. 
The instructors range from new adopters of PI to experienced Pl users. The level of success was not uniform across the 
instructors. 


Our key findings are: 
e Consistent with previous studies, a supermajority of students in all studied classes liked and would recommend PI. 
e Successful Pl implementation requires that the instructor’s motivations for using Pl are clear to students. 
e The grading policy attached to in-class Pl question responses appears to have an effect on student engagement and 

satisfaction. 

Il. RELATED WORK 

Pl is characterized by asking challenging, in class conceptual questions of students. For each question, students individually 
respond, discuss the question in small groups, and respond again based on their new understanding [1]. These questions 
should target common misconceptions and/or core course concepts. To be most effective, Pl requires other supporting 
course changes. For example, many instructors require additional preparation from students before class in order to 
make best use of limited class time. This preparation can consist of pre-lecture reading and associated quizzes [1,14] or 
clicker quizzes at the start of class[14]. Interactive classrooms, including PI classrooms, have shown significant increases in 
student learning in physics[4].In Tablel: Institution, course, and instructor characteristics. For major,(decl.) and (ant.) 
denote declared and anticipated, respectively. N/A denotes data not available. 


Identifier N-A N-B N-C E-D E-E E-F E-G 

Institution RI PUI PUI RI RI RI PUI 

Size Large Large Large Large Large Large | Small 

Public/Private Public Public Public Public Public Public | Private 

Course CS1.5 CSI CSI CSI CSO CSI CSI 

Language Java Java Java Python Alice Matlab | Java 
Times Taught this Course 6 10+ 10+ 10+ 2 | 0 
Courses Taught using Pl 0 0 0 0a 10 3 | 
Students Enrolled at End of Course 64 30 36 ISI 87 98 19 
Survey participants 62 29 13 65 87 92 15 

Percentage of CS majors he aay 50% ne rhs 1% 6.4% ne 

Freshman 3% 40% 36% >95% 34% 34% 37% 

Sophomore 40% 27% 42% N/A 45% 16% 26% 

Junior 37% 27% 13% N/A 14% 33% 22% 

Senior 21% 6% 9% N/A 8% 17% 16% 

Percentage of Students who Previously used 77% 28% 22% 3% NA 31% 25% 

Clickers 

Avg. # of Pl Questions per class 7 7 6 5 5 4 5 
Length of class(min) 75 50 75 60 80° 50 80 
(D)evelopedor (A)dopted Questions D D D D D D A 


* Instructor of course E- Dhad not taught a class using Pl, but had been part of Pl research and course development. As 
such, that instructor is considered experienced in Pl. b30 minutes each week was spent on a practice code-writing quiz. 


CS, a number of studies have reported on the success of Pl, including improved student satisfaction[5,8,10],student 
learning [7, |5],final exam grades [I1, 13],failure rates [6],and retention of majors[9].The present work provides 
additional evidence that, for both new and seasoned adopters of Pl, students widely laud the change to the course 
structure. As noted in earlier research [10], it is not always the case that PI is adopted following all recommended 
practices. The results of the present work lead us to examine ways in which PI is adopted, and we find suggestive 
evidence that student satisfaction can be significantly impacted by grading clicker results on correctness rather than 
participation. 
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Each of the seven instructors in this study reported the teaching of one introductory programming course. Four of the 
instructors had experience with Pl, either by having taught a course in Pl or by having participated significantly in the 
development of a Pl course. The other three instructors were new to PI. We label our courses with two letters: 
N(novice) or E (experienced) to indicate the instructor’s Pl experience, and a letter A-G to differentiate each 
course(assigned, within N and E categories, by decreasing years of teaching experience for the instructor). Course and 
instructor characteristics are provided in Tablel. 


A brief description of each course is provided below. Course N-A: CS1.5-Object-Oriented Design and Programming: This 
course concentrates on the object-oriented paradigm, particularly encapsulation, inheritance, and polymorphism. 
Programming assignments emphasize graphics and event-driven interaction. 

Course N-B: CS!|—Computer Science |: This course is an introduction to programming and computer science whose 
topics include: simple and structured data types, program control structures, problem analysis, algorithm design, and 
implementation using a high-level language (Java). 

Course N-C CSI-Introduction to Computing I: This Java course covers types/variables, assignment, conditions, loops, 
classes/objects, files, and arrays. 

Course E-D: CSI: This course is an introduction to procedural programming in Python for CS majors covering basic 
types, expressions, state, control structures, function definition and use, and lists. 

Course E-E CSO — Fluency in Information Technology: This non-majors computing course is required for all psychology 
majors and as a general education requirement for a subset of university students (those with in a specific “college”).The 
goals of this “general education” course in computing include computational thinking and communicating and collaborating 
about computational artifacts (in this case Alice programs and Excel sheets). 

Course E-F: CSI — Introduction to Programming in Matlab: This course is an introduction to MATLAB programming for 
the Cognitive Science department and uses the Media Computation approach [3]. Students study foundational 
programming constructs such as data manipulation, conditional statements, for-loops, while-loops, and various types of 
vector and matrix indexing. 

Course E-G:CS| — Computer Science |: This course is a required course for CS and Mathematics majors taught using 
Media Computation [3]. Concepts include variables, objects, methods, loops, conditionals, and class design. 


Table2: Student feedback on the value of Pl. Percentages reflect student agreement. Agreement values under 80% are 
highlighted. 


Question / Identifier N-A N-B N-C | E-D | E-E E-F E-G 


Thinking about clicker questions on my own, before discussing | 68% 86% | 100% | 94% | 95% | 95% | 100% 
With people around me, helped me learn the course material. 
Most of the time my group actually discusses the clicker 
question. 

The immediate feedback from clickers helped me focus on weak 
Nesses in my understanding of the course material. 
Knowing the right answer is the only important part of the 
clicker question.* 

Generally, by the time we finished with a question and 
discussion, | felt pretty clear about it. 

Clickers helped me pay attention in this course compared to 58% 93% | 100% | 95% | 90% | 90% | 100% 

traditional lectures. 
Clicker with discussion is valuable for my learning. 74% 93% | 100% | 100%| 94% | 91% | 93% 


| recommend that other instructors use this approach (reading 71% 90% 1oo% | 98% | 93% | 87% | 100% 
Quizzes, clickers, in-class discussion) in their courses. : ° ° = . ° - 


90% 93% | 100% | 88% | 98% | 97% | 100% 


74% 96% | 100% | 95% | 99% | 91% | 100% 


37% 14% 15% | 12% | 23% | 17% | 20% 


69% 90% | 100% | 94% | 97% | 84% | 93% 


One instructor, experienced in teaching Pl, is known by all instructors and actively assisted the novice instructors in 
weekly half-hour Skype meetings during their first Pl term. The instructors surveyed their students using a common 
instrument, which enabled comparison of responses across courses. 


4. RESULTS 
In student self-report surveys, we asked for views on the value of the PI approach in supporting various aspects of the 
learning experience and views on the instructor’s implementation of PI in the class room. 
4.1 Student Perception of the Value of Pl 
Students reported on their perception of the value of Pl (see Table 2). For all but one question (denoted with a *), higher 
percentages are better. Responses below 80% positive are highlighted. We note two trends in Table2. 
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The first trend is that students overwhelmingly value Pl. They report that they pay better attention in class, believe it 
helps them identify weaknesses earlier, and believe the process helps them learn. As a result, the vast majority of students 
(91% per class on average) recommend that more instructors use PI in their courses. In two classes,100 % of students 
would recommend that other instructors use Pl. These results demonstrate that across a wide range of institutions and 
instructors, students both value Pl and desire that Pl be used by more instructors. The second trend is that the students 
in Course N-A perceive PI considerably differently than students in other courses. Compared to students in other 
courses, the students in Course N-A recommend PI less often, felt discussion was less valuable, and generally reported 
less value from the Pl process. Perhaps most striking is the large percentage (37%) of students who believe the value of a 
clicker question is only in having the correct answer. We will revisit this anomalous result in the Discussion. 


4.2 Student Perception of PI Implementation 

Table 3 indicates student satisfaction with how PI was implemented regarding difficulty and timing. To express 
dissatisfaction, students could respond either “too long”/’’too difficult” or “too short’/“too easy.” The majority of classes 
saw high degrees of satisfaction with the PI implementation. Both Courses N-A and E-F stand out as having lower levels 
of satisfaction with the implementation, but recall that only Course N-A experienced the overall lower value of PI. 
Question Difficulty. In Course N-A, some students (23%) felt that questions were too difficult. Only two other courses 
had more than 4% of students who reported that questions were too difficult: Course E-F with 14% and Course E-G with 
13%. Question Time Allowed. In general, if students were unsatisfied with the time allowed for the initial vote, then they 
felt that they had too little time (notably in Courses N-A,N-C, E-D, E-E, and E-G) rather than too much time. Course N- 
A was again an outlier with 21% of students responding that they had too little time relative to 7% who felt they had too 
much time. Although instructors set the questions (and drive the pace), it is critical for students to be given time to think 
through the questions on their own. Question design (e.g. word choice, clarity, answer options) can seriously impact 
student time needed to read the question. Discussion Time Allowed. An interesting trend appears regarding time allowed 
for peer discussion. Of those students who were not satisfied with the time allowed for peer discussion in the courses of 
all four experienced instructors, more students felt that too much time was allowed. A possible reason for this is that 
experienced Pl instructors are more comfortable spending time on peer discussion, and may have personal evidence 
suggesting the value of providing students with more time to talk among themselves. For novice instructors, time 
circulating in the classroom or silently standing up-front can be initially un nerving, and at the very least is a change in 
their teaching style. However, results in Course N-A more closely match those of courses taught by experienced 
instructors in that unsatisfied students felt that too much time was allowed. As can be observed from Figure |,this may be 
related to the comparatively fewer students in Course N-A who report always discussing with their peers: if some 
students are not discussing, then they are waiting for class to move forward. 

Table3: Student feedback on the implementation of PI. Percentages reflect students responding “OK” or “About right”. 
Values under 80% are highlighted. 
Question/Identifier N-A | N-B | N-C] E-D | E-E E-F E-G 


From the point of helping me learn, the content of clicker questions 76% | 83% |100%| 88% | 94% | 78% | 80% 
was:( too hard, okay, too easy) 

In general, the instructor gave us enough time to read and understand 
The questions before the first vote: (too short, about right, too long) 
The amount of time generally allowed for peer discussion was: (too 
short, about right, too long) 

In general, the time allowed for class-wide discussion( after the group 
vote) was: (too short, about right, too long) 


72% | 89% | 92%) 81% | 87% | 78% | 87% 


89% | 87% | 92% | 79% | 86% | 77% | 73% 


70% | 86% |100%| 64% | 81% | 63% | 93% 


100% 
90% 
80% 
70% 
60% 
50% 
40% 
30% 
20% 
10% 


re 0 9 o& & 


= ~ = & ae vy ‘ 
Figure |: Student responses to the question: Which of the following describes your discussion practices this term? 
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Figure2: Student responses to the prompt: The professor explained the value of using clickers in this class. 


Class-wide Discussion Time Allowed. Regarding class-wide discussions, unsatisfied students felt that they had too much 
time in Courses N-A,E-D,E-E, and E-F. Course E-F had the most students reporting that too much time was spent on 
class-wide discussion (26% too much compared with|1% too little). This instructor had little experience with the 
programming language used in the course; it is possible that this caused a mismatch between where students struggled 
and where the instructor anticipated struggles. Student Behavior during Discussion. Figure | provides the breakdown of 
student responses regarding their discussion habits. For all but one course, the majority of students reported valuing the 
discussion with their group as it helped them learn. For all courses, only a small minority of students either did not 
discuss because they did not value discussion or because they were too shy. Explanation of the Purpose of PI. Figure 2 
provides the breakdown of student responses regarding the explanation from the instructor on why clickers were being 
used. For all but Instructor N-A, more than 90% of students thought that the instructor explained the use of clickers well 
or did so too much. Instructor N-A had a considerably larger percentage of students reporting that they were unclear 
why they were using clickers. The other anomalous course result was Course E-G, where 20% of students felt that the 
instructor explained the use of clickers too much. 

5. DISCUSSION 
5.1 Value to Students and Instructors 
Our results suggest that PI can be successfully adopted at a variety of institutions in introductory computing courses. At 
least 71% of students (avg. 91%) would recommend that other instructors use this approach in their courses. The seven 
instructors in this work all reported a dramatic change in their usual classroom experience upon implementing Pl. These 
instructors varied in terms of reasons for adoption, amount and type of adoption support, and teaching experience. 
Instructors had varying comfort levels with the courses and varying support for developing and implementing clicker 
questions. In fact, some had no more to go on than notion of allowing students to “test” their knowledge. Some read 
publications on its use, some reviewed clicker slides prepared and used in computing courses, and some had the 
opportunity to TA for or repeatedly observe an experienced PI instructor. Independent of these background factors, all 
instructors report that the next time they teach this class, they will teach it with Pl. This said, one course (N-A) stands 
out as yielding poor survey results compared to the other courses. We next reflect on why this may be the case. 


5.2 Exploring Course N-A Differences 

Differences between N-A and the other courses can be seen fairly uniformly in both the valuation and implementation 
surveys. Through reflective discussion and our reading of Pl reports in other disciplines, we highlight two structural issues 
for discussion. First, the instructor required correctness for portion of the PI grade (each question was worth 2 points: | 
for correctness and | for participation). Second, a notable 25% of the class felt that the instructor did not sufficiently 
explain why clickers were being used. Together, we posit that these two issues contributed to a very different classroom 
culture regarding Pl, as evidenced by the markedly different student survey responses. In sum, students in Course N-A 
(more than those at other institutions): 

|. Felt that clicker questions were too hard 

2. Felt that too little time was allotted to read and understand questions 

3. Did not always discuss with their peers 

4. Did not find value in hearing other students provide explanations in class-wide discussion and, 

5. Reported lesser perception of the value of PI for their learning. 


5.3 Grading on Correctness 
We suspect that the change from grading on participation to grading on correctness fundamentally changes the 
atmosphere of the course. Consider the first four points in the above list. 
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This kind of student affect feelings that questions are too hard and that discussion is not useful make sense if students 
perceive Pl questions as “standard quiz” questions rather than peer discussion questions. The fact that Instructor N-A 
gave points for correctness, even though he/she also gave points for participation, may have factored into students 
perceiving questions as a test of some-thing they should already know, rather than tools designed to build their 
understanding. There are two other concerns related to grading on correctness rather than participation. The first is that 
students become concerned about arguing their interpretation of the question, and this can disrupt the learning process. 
A focus on learning is hampered when an incorrect answer in the learning process itself has grade-based implications. The 
second, related issue, is that questions cannot be too difficult or will be viewed as unfair. This is problematic when the 
recommended correctness range for the individual vote on PI questions is 35-70% [1] and when there is evidence that 
students benefit considerably more from difficult questions [15].Instructors who both grade on correctness and recognize 
the limits that this poses on difficulty may respond in-kind by offering easier questions. This shying away from difficult 
questions may explain why many students thought that PI was not beneficial for their learning. We can see evidence of 
students experiencing Course N-A differently than the other courses through open-ended student responses. For 
example, a student in Course N-A reports: 


In this lecture | am more focused on trying to guess the answers to the questions than on internalizing and under-standing 
the course content. In standard lectures | am focused on taking clear and thorough notes and absorbing the material. 


Student in Course N-A 

A quote from a student in Course E-G may provide more insight into the importance of the grading structure, based on 
their report of clicker use in a different class: 

| have another clicker course, this one is far better. In the other course, every clicker question is graded. It feels too 
much like the professor just wants to play game show host and puts to [sic] much weight on the correct answers and not 
the process of getting the answer. This is opposite for this course. | feel like participation should just be graded. 

Student in Course E-G 

Given this criticism of grading on correctness, it is essential to examine Course E-E. Like Course N-A, Course E-E had a 
correctness requirement, but it was differently implemented in a manner that reinforced the role of clicker questions in 
the learning process rather than the assessment process. In Course E-E, students we regraded on participation, but they 
had to get at least 50% of all questions (including individual and group responses) in each lecture period correct to get 
those participation points. This approach was devised based on experience in an earlier offering of the course, which 
required only participation and where students were noted answering randomly and engaging in unrelated activities in the 
classroom. The instructor believed that a change in policy was needed based on the fact that the course was a required, 
non-majors course. Students generally did not start the course with a great deal of enthusiasm, nor much understanding 
of what value the course held for them. In explaining the “half correct” policy, the instructor was able to reiterate that 
clicker questions are for helping students engage in developing expert analysis and argumentation skills. The policy 
reinforced Instructor E-E’s primary course learning goal: getting students to learn how computing people see problems. 

A student from Course E-E discusses this policy, noting his/her need to prepare, but expressing satisfaction with the 
awarding of discussion points: 

This class was very different from my other classes as it truly made me be on top of my game. | did like the grading 
structure and how participation points were fair. 

Student in Course E-E 

And some students did appear to get the message that their engagement in reasoning about the question, not simply 
getting points, was the goal. [In this course], discussion and proof of understanding is a vital part of lecture. With clicker 
questions, as a student, | was able to engage in thoughtful reasoning. 

Student in Course E-E 

5.4. Explaining Pedagogical Change 

It is critically important to explain (repeatedly)to students any deviation from expected classroom norms. Students have 
both experience with and expectations of college classroom learning. They know what happens in a lecture and have 
techniques that they expect to use in order to learn and measure their progress toward success. From our collective 
experience, we can report that students claim that they “have to sit at the front of the class” in order to learn/stay awake, 
want the instructor to “just explain it,’ or complain that the lack of lecture “forced me to learn it all myself.” Pl 
completely pulls the rug out from under the students by challenging them to reexamine their established, comfortable, 
and often perceived successful learning habits. 


In a popular 2-page “Tips for Successful Clicker Use” summary, Douglas Duncan (Univ. of Colorado, Astronomy) lists as 
his second tip:“You MUST MUST MUST explain to students why you are using clickers. If you don’t, they often assume 
your goal is to track them like Big Brother, and force them to come to class. Students highly resent this.’(emphas is 
original)[12].Perhaps more tellingly, the first item on his list of Practices that Lead to Failure is “|. Fail to explain why you 
are using clickers.”[2] 
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In Course N-A,74% of students report that their instruct or explained to the class why he/she was using clickers. How- 
ever, the remaining 26% of students who felt that clickers were not explained adequately (or at all) was the highest 
percentage among the courses. This emphasizes the importance of explaining the value of PI not just once and in not just 
one way. Moreover, for Course N-A, it is likely that students were especially sensitive to this issue as most students had 
previously taken an introductory course where the instructor used clickers to take attendance. The presence of these 
kinds of non-pedagogical uses of clickers only heightens the need to explain the pedagogical goals of PI. 

5.5 Other Factors 

Note that, in addition to the two structural issues we have highlighted, Course N-A also differs from other courses in 
other ways. The course is aCSI.5 course (not aCSI course), many of the students used clickers in the past, and the 
proportion of freshmen is lower. There is little precedence for these differences contributing to the outlying survey 
results for this course, however. The earliest experience reports of Pl in CS report on successful adoption in CSI.5 
courses [10], and we know of no evidence suggesting that any novelty of clickers wears off after a single course. 


CONCLUSION 

In this multi-institutional study of student satisfaction in Peer Instruction (Pl) courses, we find further evidence of PI being 
highly valued by students. We also find that one course yielded lower student satisfaction than the other courses. We 
have argued that this lower satisfaction may have stemmed from two adoption decisions: grading on correctness, and not 
convincingly arguing to students why clickers and PI are being used. We offer two conclusions here. First, new adopters 
of PI can expect levels of success similar to those reported by others with considerable Pl teaching and development 
experience. Second, it is important to evaluate adoption decisions. Pedagogy so widely-applicable as Pl will inevitably 
engender debate over the particulars of day-to-day implementation and interaction with students. We encourage all Pl 
instructors to reflect on and make explicit the reasons undergirding their Pl-based decisions so as to maximize the value 
of PI for students. 
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